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Introduction 

This  report  summarizes  work  conducted  during  the  period  from  February  15,  1987  to  July 
31,  1990  on  the  psychology  of  decision-making  processes  and  provides  a  chronological  sequel  to 
an  earlier  report  covering  the  period  from  October  15,  1983  to  February  14,  1987  (Einhom  & 
Hogarth,  1987).  The  structure  of  this  report  reflects  the  three  major  topics  under  which  research 
has  been  conducted.  However,  a  discussion  of  these  topics  is  prefaced  by  some  general  remarks 
emphasizing  the  central  theme  underlying  the  work  and  the  research  strategy  adopted. 

Both  the  central  theme  of  the  work,  as  well  as  the  general  research  strategy,  are  the  same  as 
that  outlined  in  the  previous  report  (Einhom  &  Hogarth,  1987).  They  are  therefore  restated  here. 

Central  theme.  The  central  theme  underlying  much  of  the  work  is  that  complex  judgments 
and  decisions  result  from  simple  psychological  processes  that  interact  with  highly  variable  and 
complex  environments.  In  other  words,  whereas  basic  psychological  processes  are  quite  simple, 
the  complexity  of  behavior  results  from  the  way  such  processes  interact  with  the  complexities  of 
the  environment  (see  Hoganh,  1986).  However,  it  is  the  complexity  of  behavior  that  we  seek  to 
elucidate. 

Research  strategy.  Given  the  above,  the  research  strategy  adopted  has  sought  to  develop 
descriptive,  quantitative  models  of  judgment  and  choice  by: 

1.  Identifying  basic  psychological  processes. 

2.  Developing  parsimonious  mathematical  models  of  these  processes. 

3.  Constructing  tasks  and  environments  in  which  to  test  the  predictions  of  the  models. 

4.  Empirically  testing  the  predictions. 

Areas  of  research.  Research  has  been  conducted  in  three  areas.  These  are  (a)  the 
psychology  of  choice  with  particular  reference  to  decisions  made  under  conditions  of  ambiguity 
and  ignorance;  (b)  belief  updating;  and  (c)  the  effects  of  exactingness1  and  incentives  on  learning. 
The  following  three  sections  of  this  report  discuss  each  of  these  areas  of  research.  In  each  section. 


The  term  “exactingness”  is  explained  below. 
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we  first  outline  the  issues  motivating  the  research  and  the  principles  underlining  our  approach  to 
the  specific  problems.  After  presenting  the  models  developed  for  each  project,  we  summarize 
implications  and  the  major  empirical  results  obtained  to  date. 


Venture  Theory  and  Choice  under  Ambiguity  and  Ignorance 

One  dimension  on  which  laboratory  studies  of  decision  making  lack  ecological  validity  is 
that  subjects  are  required  to  make  choices  between  options  for  which  precise  probabilistic 
information  is  presented.  For  example,  subjects  are  asked  to  make  choices  between,  say,  winning 
$2,000  with  probability  .8  (and  nothing  with  probability  .2),  versus  a  .7  chance  of  winning 
$2,400  (and  nothing  with  probability  .3).  In  the  real  world,  however,  it  is  rare  that  people  face 
such  precise  choice  alternatives.  Instead,  if  probabilistic  information  is  provided,  at  best  it  is  only 
likely  to  be  vaguely  specified,  e.g,  “ about  8  chances  in  10  of  winning,”  or  “between  70%  and 
90%. ”  At  worst,  it  is  not  provided  and  chances  have  to  be  inferred  from  the  context  by  the 
decision  maker.  In  previous  work  (notably  Einhorn  &  Hogarth,  1985),  we  developed  a 
psychological  model  of  how  people  take  account  of  vague  probabilistic  information  when 
estimating  probabilities.  Research  conducted  under  this  heading  during  the  repoiting  period 
extended  the  earlier  work  in  three  ways.  First,  additional  experimental  tests  of  the  ambiguity  model 
were  conducted  using  scenarios  that  mimicked  real-world  situations.  Second,  the  psychological 
principles  behind  the  ambiguity  model  were  generalized  in  order  to  develop  a  general  model  of 
choice  which  would  also  handle  situations  where  probabilities  are  known  with  precision.  Third,  a 
new  line  of  investigation  was  begun  involving  situations  where  no  (even  vague)  probabilistic 
information  was  available  to  subjects.  To  discuss  these  developments,  it  is  first  appropriate  to 
provide  an  overview  of  the  ambiguity  model. 

Overview  of  ambiguity  model 

The  main  psychological  assumption  underlying  the  model  is  that  the  subjective  weights 
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given  to  ambiguous  probabilities  are  the  end  result  of  a  mental  anchoring-and-adjustment  process. 
People  are  assumed  to  anchor  on  a  particular  estimate  of  the  probability  and  then  adjust  this  by 
imagining,  via  a  mental  simulation  process,  other  values  that  the  probability  could  take.  To 
illustrate,  consider  a  situation  in  which  you  are  concerned  about  the  chances  of  an  accident 
occurring  in  a  new  industrial  facility.  A  study  conducted  by  technical  experts  assesses  the  risk  as 
p  =  .001,  but  they  have  doubts  about  the  precision  of  this  estimate.  In  the  process  assumed  here,  it 
is  postulated  that  you  would  first  anchor  on  a  given  value  of  probability  (e.g.,  the  .001  provided 
by  the  experts)  and  then  imagine  or  "try  out"  other  values  the  probability  could  take,  both  below 
and  above  the  anchor.  Depending  on  the  circumstances  (see  below),  you  would  not  necessarily 
accord  equal  weight  in  imagination  to  possible  values  of  the  probabilities  on  both  sides  of  the 
anchor.  For  instance,  in  the  present  example  values  above  the  anchor  may  well  weigh  more 
heavily  in  imagination  than  those  below  (the  occurrence  of  accidents  might  be  salient).  The 
resulting  weight  given  to  the  ambiguous  probability  is  taken  to  reflect  both  the  initial  anchor  and  the 
net  effect  of  the  mental  simulation  process  and  can  be  written 

S(Pa)  =  PA  +  (kg  -  ks)  (1) 

where  p  a  is  the  anchor,  kg  represents  the  values  and  weight  accorded  in  the  mental  simulation  to 
values  of  p  greater  than  the  anchor,  and  ks  corresponds  to  the  weighted  values  below  the  anchor. 

To  make  these  notions  operational,  one  needs  to  specify  (1)  how  the  anchor,  p^,  is 
established,  (2)  what  affects  the  amount  of  mental  simulation  (i.e.,  the  ranges  of  alternative" 
probability  values  considered),  and  (3)  what  determines  the  sign  or  direction  of  the  adjustment 
process. 

(1)  In  ambiguous  circumstances,  some  initial  value  of  the  probability  is  assumed  to  be 
typically  available  to  the  decision  maker.  This  may  be  a  figure  based  on  historical  data,  provided  by 
experts  (as  in  the  example  above),  or  selected  from  memory. 

(2)  If  the  decision  maker  has  sufficient  knowledge  to  assign  a  unique  value  to  the 
probability  there  would  be  little  or  no  mental  simulation.  When  the  probability  is  ambiguous,  one 
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would  expect  considerable  simulation,  the  extent  of  which  is  assumed  to  be  positively  related  to  the 
amount  of  perceived  ambiguity. 

(3)  The  sign  of  the  adjustment  process  is  determined  by  the  person's  attitude  toward 
ambiguity.  This  could  reflect  personal  dispositions  toward  optimism  or  pessimism,  but  is  assumed 
to  depend  on  situational  variables  such  as  the  sign  or  size  of  outcomes  or  whether  the  context  of  the 
situation  induces  caution  (as  when  considering  insurance)  or  playfulness  (as  when  gambling). 

The  manner  in  which  imagination  affects  the  anchor  value,  p^,  in  the  ambiguity  model  can 
be  shown  by  depicting  the  judgmental  compromise  that  results  from  the  anchoring-and-adjustment 
process  as  a  function  of  the  anchor  probability.  This  is  illustrated  in  the  three  panels  of  Figure  1 
below. 

In  interpreting  the  panels  of  Figure  1 ,  recall  that  two  forces  cause  the  final  judgment  to 
deviate  from  the  anchor.  These  are  the  amount  of  perceived  ambiguity  and  the  person's  attitude 
toward  ambiguity  in  the  circumstances.  The  former  determines  the  amount  of  mental  simulation 

and  thus  the  extent  to  which  the  ambiguity  function  deviates  from  the  diagonal  (45°)  line;  the  more 
the  perceived  ambiguity,  the  greater  the  deviation.  The  latter  determines  the  direction  of  the 
adjustment  and  thus  the  point  at  which  the  ambiguity  function  crosses  the  diagonal. 


<*>  Q>)  (c) 


Figure  1;  Examples  of  three  “ambiguity”  functions 
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Consider  first  the  extreme  anchors  of  =  0  and  =  1.  In  both  cases,  the  adjustment 
can  only  be  in  one  direction,  up  for  p^  =  0  and  down  for  p^  =  1,  thereby  illustrating  the  fact  that 
the  location  of  p^  places  constraints  on  the  ranges  of  values  that  can  be  imagined  above  and  below 
the  anchor.  Thus  S(p^)  >  p^  when  p^  =  0  and  S(p^)  <  p^  when  p^  =  1.  In  general,  S(p^)  will 
overweight  small  probabilities  and  underweight  large  ones;  what  changes  from  situation  to 

situation  is  the  point  at  which  the  ambiguity  function  crosses  over  the  diagonal  (45°)  line,  i.e., 

where  overweighting  changes  to  underweighting.  In  Figure  la,  values  of  the  probability  below  the 
anchor  are  weighed  in  imagination  more  heavily  than  those  above,  and  the  cross-over  point  lies 
below  pA  =  .5.  In  Figure  lb,  values  above  the  anchor  are  weighted  more  heavily  than  those  below. 
Here  the  cross-over  point  lies  above  .5.  In  Figure  lc,  values  above  and  below  the  anchor  are 
weighted  equally  such  that  the  crossover  occurs  at  .5. 

To  summarize,  the  ambiguity  function  shows  overweighting  of  small  anchor  values  but 
underweighting  of  larger  ones.  The  point  at  which  the  function  changes  from  over-  to 
underweighting  depends  on  the  person's  attitude  toward  ambiguity.  For  example,  assuming  that 
people  are  generally  cautious  in  the  face  of  risk,  the  ambiguity  function  would  resemble  that  shown 
in  Figure  la  if  the  decision  maker  is  concerned  with  the  possibility  of  obtaining  a  positive  outcome. 
On  the  other  hand,  when  faced  with  the  possibility  of  a  loss  (e.g.,  when  assessing  the  risk  of  a 
new  technology),  the  function  would  be  better  represented  by  Figure  lb.  This  is  because  caution 
induces  greater  concern  for  possible  values  of  the  probability  lying  below  rather  than  above  the 
anchor  in  the  case  of  potential  gains,  whereas  the  contrary  holds  for  losses.  We  also  argue  that  the 
location  of  the  cross-over  point  will  be  affected  by  the  degree  of  caution  engendered  by  the 
situation.  Thus,  when  facing  the  ambiguous  chance  of  gaining  a  very  large  sum  of  money,  the 
cross-over  point  will  be  closer  to  p^  =  0  than  in  a  case  where  a  small  sum  is  involved.  Similarly, 
when  faced  with  a  large  potential  loss,  the  cross-over  point  will  be  closer  to  p^  =  1  than  in  a 
situation  involving  a  small  loss. 

As  noted  in  the  previous  report,  this  model  handles  many  of  the  paradoxes  of  choice  theory 
and,  in  particular,  that  posed  by  Ellsberg. 
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Additional  empirical  results 

In  work  completed  during  the  period  under  review,  further  features  of  the  model  were 
exploited: 

(1)  Further  analyses  were  completed  of  a  data  base  of  questionnaires  collected  from 
practicing  actuaries  --  see  Hogarth  and  Kunreuther  (1989;  1990).  The  results  of  these  studies  were 
substantially  in  agreement  with  the  theoretical  predictions  of  the  ambiguity  model  (both  in  terms  of 
quantitative  outputs  and  some  process  details),  and  provided  important  information  concerning  the 
conditions  under  which  ambiguity  does  and  does  not  affect  the  relative  thickness  of  insurance 
markets.  In  addition  to  providing  these  substantive  conclusions  concerning  the  market  for 
insurance,  these  results  are  important  in  that,  because  the  subjects  were  professionals  operating  in 
their  area  of  expertise,  they  speak  to  the  ecological  validity  of  the  model. 

(2)  Experimental  tests  of  the  model  were  extended  to  scenarios  involving  bargaining  over 
terms  in  the  purchase  of  industrial  goods  (using  business  executives  as  subjects),  and  the  manner 
in  which  ambiguity  might  affect  the  tendency  for  different  parties  to  a  lawsuit  to  either  settle  out  of 
court  or  go  to  trial.  In  both  of  these  situations,  predictions  were  made  by  recognizing  that 
ambiguity  would  differentially  affect  estimations  of  uncertainty  made  by  persons  with  different 
roles  in  a  transaction  or  dispute  (e.g.,  plaintiffs  vs.  defendants).  This  occurs  largely  because 
different  roles  can  “frame”  the  probabilities  of  the  same  events  as  involving  potential  gains  or 
losses  —  see  Figure  1.  The  differential  predictions  of  the  model  were  borne  out  by  the 
experimental  results  --  see  Hogarth  (1989b). 

Venture  theory 

Success  with  the  ambiguity  model  suggested  that  the  proposed,  underlying  cognitive 
processes  might  also  apply  to  judgments  and  choice  in  cases  for  which  ambiguity  was  absent.  This 
insight  led  to  the  development  of  venture  theory  (Hogarth  &  Einhom,  1990a). 

Venture  theory  is  a  generalization  of  the  well-known  prospect  theory  model  developed  by 
Kahneman  and  Tversky  (1979).  The  generalization  takes  the  form  of  providing  an  explicit 
psychological  model  for  the  prospect  theory  decision  weight  function.  In  brief,  the  model  assumes 
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that,  even  in  the  presence  of  known  probabilities,  people  will  engage  in  a  mental  simulation 
process  similar  to  that  assumed  in  the  ambiguity  model.  Moreover,  because  this  simulation  is 
affected  by  both  the  sign  and  size  of  payoffs,  the  “decision  weight"  function  will  differ  depending 
on  the  values  of  both  of  these  variables  (see  also  description  of  the  ambiguity  model,  above). 
When  these  assumptions  concerning  the  decision  weight  function  (for  details  see  Hogarth  & 
Einhorn,  1990a)  are  combined  with  the  properties  of  the  prospect  theory  value  function,  it  is 
possible  to  obtain  a  series  of  predictions  concerning  effects  on  attitudes  toward  risk  and  ambiguity 
that  are  induced  by  different  levels  of  both  probabilities  and  payoffs  as  well  as  the  sign  of  the  latter. 
These  predictions  are  important  in  that  they  indicate  the  conditions  under  which  payoffs  and 
probabilities  do  and  do  not  interact  in  their  effects  on  revealed  attitudes  toward  risk  and  ambiguity. 
Indeed,  venture  theory  provides  one  of  the  first,  explicit,  theoretical  treatments  of  this  issue  that  is 
derived  from  psychological  as  opposed  to  mathematical/economic  reasoning. 

Venture  theory’s  predictions  were  tested  in  a  series  of  three  experiments  (see  Hogarth  & 
Einhorn,  1990a).  Two  of  the  experiments  involved  choices  using  hypothetical  gambles  and  one 
involved  gambles  with  real  payoffs.  One  interesting  result  of  the  experiments  was  that  although 
almost  all  of  venture  theory’s  14  predictions  were  validated  in  the  experiment  involving  real 
payoffs,  for  the  hypothetical  payoffs  the  rate  of  agreement  was  not  so  high.  In  particular,  attitudes 
toward  risk  and  ambiguity  seem  to  differ  in  the  domain  of  losses  when  payoffs  are  real  as  opposed 
to  hypothetical.  Parenthetically,  the  two  venture  theory  predictions  that  were  not  upheld  in  the  real 
payoff  experiment  concerned  the  nature  of  probability  x  payoff  interactions  in  attitudes  toward  risk 
and  ambiguity  for  losses.  That  is,  although  these  interactions  were  observed,  their  precise  form 
was  not  that  predicted  by  venture  theory. 

Finally,  the  above  tests  of  venture  theory  suffer  from  the  fact  that  they  occur  in  the  confines 
of  the  psychological  laboratory.  Unfortunately,  an  important  cost  of  doing  research  on  risky 
decision  making  is  that  any  new  models  proposed  must  be  shown  to  be  able  to  explain  the  large 
experimental  literature  even  though  many  researchers  now  believe  that  the  gambling  metaphor  of 
risky  decision  making  is  incomplete,  and  that  work  must  be  conducted  in  more  ecologically 
relevant  settings.  The  advantage  of  venture  theory  is  that  it  explicitly  models  how  key 
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psychological  constructs  of  emotion  (e.g.,  caution)  and  cognition  (e.g.,  imagination)  affect 
decision  weights.  It  therefore  offers  a  useful  structure  for  studying  the  impact  of  uncertainty  both 
in  and  outside  the  psychological  laboratory. 

Decision  making  under  ignorance 

The  domain  of  venture  theory  is  decisions  under  risk  and  ambiguity.  But,  how  do  people 
make  decisions  when  they  have  no  or  very  little  information  about  probabilities  or  payoffs,  i.e., 
under  conditions  of  ignorance?  For  example,  imagine  that  you  have  just  purchased  a  VCR.  The 
salesperson  now  asks  whether  you  would  like  to  buy  an  extended  warranty  agreement  for,  say, 
$40.  How  do  you  evaluate  this  decision  if  (a)  you  have  little  idea  of  the  chance  of  incurring  a 
breakdown  nor  (b)  how  much  repairs  would  cost? 

One  way  people  might  handle  this  kind  of  situation  could  be  to  generate  arguments  with 
themselves  that  would  either  favor  or  be  against  buying  the  warranty  (or  both).  For  example,  one 
argument  might  be  “Don’t  buy  because  these  are  not  usually  good  deals.”  Another  could  be  “Buy 
because  if  you  don’t,  you  might  regret  it  in  the  case  of  a  breakdown.”  * 

Our  latest  work  (conducted  together  with  Howard  Kunreuther  at  the  University  of 
Pennsylvania)  has  begun  to  explore  “decision  making  under  ignorance”  using  this  arguments 
framework.  So  far,  we  have  conducted  two  preliminary  studies  using  the  warranty  purchase 
decision  as  a  paradigmatic  case.  The  basic  experimental  design  involves  presenting  subjects  with 
warranty  scenarios  and  then  asking  for  both  a  decision  (buy  or  not  buy)  and  the  arguments  used  in 
reaching  the  decision  (either  in  a  “free”  format  or  by  checking  off  a  list  of  preestablished 
arguments).  Our  preliminary  results  may  be  summarized  briefly  as  follows: 

(1)  people  try  to  focus  on  numbers  in  the  problems  if  they  can.  For  example,  most  subjects 
seem  to  compare  the  cost  of  the  warranty  to  the  cost  of  the  nroduct. 

(2)  people’s  arguments  in  these  situations  are  quite  close  to  what  one  would  expect  from  a 
rational  analysis  of  the  problem.  For  example,  arguments  are  frequently  centered  on  assessing  the 
likelihood  of  breakdown  and/or  involve  tradeoffs,  e.g.,  feelings  of  “risk”  versus  cost  of  warranty. 

■  (3)  most  arguments  are  focussed  on  evaluating  specifics  of  the  particular  cases  as  opposed 
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to  involving  more  general  principles  such  as  “I  never  buy  these  kinds  of  warranties.” 

More  work  on  the  topic  of  the  use  of  arguments  is  currently  under  way. 

Belief  Updating 

The  previous  report  (Einhom  &  Hogarth,  1987)  described  work  on  a  model  of  belief 
updating.  There  has  been  considerable  progress  on  this  work  during  the  reporting  period.  Because 
this  work  is  fully  detailed  in  Hogarth  and  Einhom  (1990b),  this  section  only  summarizes  the  main 
points. 

The  theory  of  belief  updating  represents  a  concise,  descriptive  account  of  the  manner  in 
which  beliefs  are  updated  in  response  to  information  that  is  received  sequentially.  To  validate  this 
model,  we  have  chosen  to  focus  on  the  phenomena  of  order  effects,  i.e.,  the  fact  the  order  in 
which  information  is  received  can  affect  the  level  of  the  final  opinion.  These  order  effects  are  seen 
as  arising  from  the  interaction  of  information-processing  strategies  and  task  characteristics.  Based 
on  an  extensive  review  of  the  literature  (involving  some  60  different  papers),  the  following  task 
variables  were  identified  as  key:  (1)  complexity  of  the  stimuli  processed  for  each  revision  of  belief; 
(2)  length  of  the  series  of  evidence  items  processed;  and  (3)  response  mode  (whether  responses  are 
made  in  response  to  each  piece  of  evidence,  i.e.,  Step-by-Step,  or  only  after  all  evidence  has  been 
processed,  i.e.,  End-of-Sequence).  The  model  itself  assumes  that  people  handle  belief-updating 
tasks  by  a  general,  sequential  anchoring-and-adjustment  process  in  which  current  opinion,  or  the 
anchor,  is  adjusted  by  the  impact  of  succeeding  pieces  of  evidence. 

In  algebraic  terms,  the  model  can  be  written 

Sk  =  Sk-1  +  wk  [s<xk)  -  R]  (2) 

where  =  degree  of  belief  in  some  hypothesis,  impression  or  attitude  after 

evaluating  k  pieces  of  evidence  (0  <  <  1). 
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Sk-1  = 
s(xk)  = 


R 


wk  = 


anchor  or  prior  opinion.  The  initial  strength  of  belief  is  denoted  Sq. 
subjective  evaluation  of  the  kth  piece  of  evidence.  (Different  people 
may  accord  the  same  evidence,  xk,  different  evaluations), 
the  reference  point  or  background  against  which  the  impact  of  the 
kth  piece  of  evidence  is  evaluated. 

the  adjustment  weight  for  the  kth  piece  of  evidence  (0  <  wk  <  1). 


Within  this  general  model  there  are  three  important  subprocesses  which  concern  (a)  how 
evidence  is  encoded  --  this  can  be  done  in  two  ways,  either  as  a  deviation  relative  to  the  size  of  the 
preceding  anchor  (operationalized  by  setting  R  =  Sk.j)  or  as  positive  or  negative  vis-a-vis  the 
hypothesis  under  consideration  (operationalized  by  setting  R  =  0).  The  former,  labeled  estimation 
mode,  results  in  data  consistent  with  averaging  models  of  judgment;  the  latter,  labeled  evaluation 
mode,  implies  adding  models;  (b)  how  evidence  is  processed  -  whether  beliefs  are  revised  in 
response  to  each  piece,  of  evidence  (i.e.,  a  Step-by-Step  strategy)  or  only  after  all  the  evidence  has 
been  processed  (i.e.,  an  End-of-Sequence  strategy),  and  (c)  how  the  adjustment  is  accomplished- 
a  contrast  assumption  is  adopted  in  which  wk  is  taken  to  be  proportional  to  Sk_  j . 

The  theory  specifies  conditions  under  which  (a)  evidence  is  encoded  in  estimation  or 
evaluation  modes,  and  (b)  use  is  made  of  the  Step-by-Step  or  End-of-Sequence  processing 
strategies.  This  analysis  is  shown  to  account  for  much  existing  data  and  to  make  novel  predictions 
for  combinations  of  task  characteristics  where  current  data  are  sparse. 

Whereas  several  experiments  on  order  effects  had  been  conducted  at  the  time  of  the 
previous  report  (Einhorn  &  Hogarth,  1987),  one  important  new  experiment  reported  in  this  paper 
dealt  with  a  manipulation  of  the  manner  in  which  evidence  was  encoded.  This  experiment  showed 
that  when  the  same  evidence  was  presented  in  alternative  forms,  intended  to  evoke  alternatively  the 
estimation  and  evaluation  modes,  order  effects  were  affected.  We  believe  this  is  a  particularly 
important  result  for  two  reasons:  (1)  whereas  there  has  been  considerable  confusion  in  the  literature 
concerning  whether  people  use  averaging  or  adding  models  of  judgment,  our  theory  explains  how 
the  same  processes  can  produce  both  types  of  data  depending  on  how  information  is  encoded. 
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Moreover,  in  some  cases  order  effects  are  induced  by  the  manner  in  which  information  is  encoded 
as  opposed  to  the  manner  by  which  it  is  processed;  and  (2)  we  have  also  provided  an  explicit 
experimental  demonstration  of  our  theoretical  hypothesis. 

Exactingness  and  Incentives 

Learning  from  feedback  is  a  critical  dimension  of  decision  making.  Feedback,  however,  is 
often  ambiguous.  In  particular,  we  note  that  feedback  from  the  outcomes  of  decisions  can  serve 
two  functions  that  are  often  confounded.  One  function  is  inferential.  Feedback  informs  the  decision 
maker  about  the  structure  of  the  underlying  task.  For  example,  when  a  student  writes  a  paper, 
feedback  in  the  form  of  a  grade  provides  information  about  how  to  write  a  good  paper.  The  second 
function  is  evaluative.  To  continue  the  example,  feedback  also  provides  information  about  the 
student’s  performance.  Was  this  good  or  bad?  Note,  however,  that  the  feedback  --  in  this  case  a 
grade  -  is  confounded.  To  what  extent  does  the  grade  reflect  the  student’s  ability  to  write  papers, 
and  to  what  extent  does  it  reflect  the  teacher’s  grading  policy? 

Evaluation  of  decision-making  performance  can  differ  on  a  dimension  that  we  term  the 
exactingness  of  the  environment  and  which  reflects  the  severity  of  penalties  imposed  for  errors. 
Tasks  are  exacting  to  the  extent  that  deviations  from  optimal  decisions  are  heavily  punished,  and 
lenient  to  the  extent  that  they  are  not. 

In  addition  to  exactingness,  decision-making  tasks  can  vary  in  the  extent  to  which  different 
levels  of  performance  have  consequences  for  the  decision  maker.  In  the  case  of  the  student  essay, 
for  example,  the  student  may  or  may  not  perceive  the  grade  as  consequential  (e.g.,  by  affecting 
chances  of  admission  to  graduate  school).  In  other  words,  tasks  can  vary  in  the  extent  to  which 
decision  makers  have  incentives  to  perform  well. 

The  present  work  examined  the  effects  of  both  exactingness  and  incentives.  There  are 
several  reasons  for  studying  these  phenomena  together.  First,  to  know  when  and  how 
exactingness  and  incentives  affect  learning  is  important  at  a  practical  level.  In  business  or  the 
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military,  for  example,  what  levels  of  exactingness  implied  by  different  evaluation  schemes  promote 
efficient  learning?  Do  real  consequences  in  terms,  say,  of  money  or  lives  help  people  learn  to  make 
decisions  more  effectively?  If  exactingness  and/or  incentives  are  detrimental,  how  can  learning  be 
structured  to  overcome  these  impediments?  Second,  despite  the  importance  of  exactingness  in 
many  real-world  tasks,  little  theoretical  attention  has  been  directed  toward  understanding  its  effects. 
Third,  and  also  from  a  theoretical  viewpoint,  controversy  exists  as  to  whether  incentives 
necessarily  improve  performance.  From  naive  behaviorist  or  economic  viewpoints,  for  example, 
one  could  argue  that  incentives  will  always  improve  performance  and  much  evidence  is  consistent 
with  this  contention.  However,  there  is  also  evidence  that  under  some  conditions  incentives  can  be 
detrimental. 

Theoretical  analysis .  We  hypothesize  that  exactingness  induces  forces  that  have  both 
positive  and  negative  effects  on  performance  when  learning  to  make  decisions  in  a  repetitive  task. 

The  positive  aspect  of  increases  in  exactingness  lies  in  the  opportunities  it  provides  for 
learning.  To  see  this,  imagine  a  situation  which  is  perfectly  “lenient.”  In  this  case,  people  always 
receive  perfect  outcomes  no  matter  what  decisions  they  take.  They  can  therefore  never  learn  how 
the  actions  they  take  are  related  to  objective  criteria.  As  the  environment  becomes  more  exacting, 
however,  outcomes  become  more  sensitive  to  “errors”  in  decisions  thereby  providing  greater 
possibilities  for  learning.  Learning,  however,  would  not  be  expected  to  increase  linearly  with 
exactingness.  Instead,  we  hypothesize  that  the  positive  aspect  of  learning  (as  measured  by 
performance)  is  an  increasing,  concave  function  of  exactingness. 

The  main  negative  aspect  associated  with  increases  in  exactingness  lies  in  the  interpretation 
of  feedback  and  subsequent  reactions  to  this.  Specifically,  as  exactingness  increases,  feedback  in 
is  increasingly  liable  to  be  negative  and,  in  the  absence  of  alternative  points  of  reference,  perceived 
as  such.  In  learning  environments,  people  are  likely  to  react  differently  to  positive  and  negative 
feedback.  Whereas  positive  feedback  reinforces  maintaining  and  refining  existing  behavior  or 
response  strategies,  negative  feedback  encourages  shifting  strategies  and  seeking  alternatives  that 
may  work  better.  Because  the  subset  of  response  strategies  that  “work”  in  exacting  environments  is 
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much  smaller  than  those  that  don’t,  continual  shifting  of  strategies  results  in  lower  performance  - 
at  least  in  the  short  run.  We  hypothesize  that  as  exactingness  increases,  the  rate  at  which  this 
negative  factor  affects  learning  does  not  decrease.  Thus,  the  negative  aspect  of  learning  (as 
measured  by  performance)  is  a  non-concave  decreasing  function  of  exactingness. 

We  propose  that  incentives  accentuate  both  the  positive  and  negative  forces  of  exactingness. 
More  specifically,  when  feedback  is  generally  positive,  as  in  lenient  environments,  incentives  will 
induce  more  consistent  application  of  apparently  successful  response  strategies  and  performance 
will  improve.  When  feedback  is  generally  negative,  as  in  exacting  environments,  incentives  will 
induce  more  intensive  search  for  alternatives,  and  performance  will  degrade,  at  least  in  the  short 
run. 

A  formal  model.  To  clarify  implications  of  the  above  arguments,  we  use  the  heuristic 
device  of  a  simple,  algebraic  model.  Let 


7t  =  k  [b  -  c  a]  (3) 

where  k  represents  performance,  k  is  a  constant  of  proportionality,  a  represents  exactingness  (a  > 
0),  b  and  c  are  coefficients  (b,  c  >  0)  representing  the  extent  to  which  the  presence  of  incentives 
accentuates,  respectively,  the  positive  and  negative  aspects  of  exactingness  (a)  on  performance  (h), 
and  X  (0  <  X  <  1)  determines  the  degree  of  concavity  of  the  function  that  represents  the  positive 
aspect  of  exactingness  (a)  on  performance. 

We  draw  two  general  implications  from  this  model.  First,  the  form  of  Equation  3  is  such 
that  performance  will  be  a  single-peaked  (inverted-U  shaped)  function  of  exactingness  (a).  This 
means  that  performance  will  be  better  when  exactingness  (a)  is  at  intermediate  rather  than  extreme 
values. 

Second,  we  can  enquire  about  how  incentives  interact  with  exactingness.  To  do  so,  assume 
that  Equation  3  represents  performance  with  no  incentives  and  denote  performance  with  incentives 
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by 


it'  =  k  [b'a^  -  c'cx] 


(4) 


where  b'  >  b  and  c'  >  c.  Next,  ask  when  performance  with  incentives  exceeds  that  without,  i.e., 
when  7t'>7t.  Simple  algebraic  manipulation  leads  to  the  condition 


(b'  -  b)/(c'  -  c)  >  a  ^ 


(5) 


The  general  implication  of  (5)  is  that  there  is  a  critical  value  of  exactingness  (a)  below  which 
incentives  lead  to  superior  performance  but  above  which  incentives  are  dysfunctional. 

Predictions.  The  model  implied  by  Equation  3  and  its  underlying  assumptions  lead  to 
several  predictions  concerning  observed  performance  (x). 

Prediction  1:  Environments  characterized  by  intermediate  levels  of  exactingness  (a)  will 
lead  to  better  performance  (7t)  than  lenient  or  exacting  environments.  (This  is  implied  by  the  fact 
that  7t  is  a  single-peaked  or  inverted-U  shaped  function  of  a. ) 

Prediction  2:  There  will  be  an  interaction  between  incentives  and  exactingness.  Whereas 
incentives  will  lead  to  improved  performance  in  lenient  environments,  they  will  become  less 
beneficial  as  exactingness  increases. 

Prediction  3:  If  the  negative  effects  of  exactingness  (a)  on  performance  (n)  are  eliminated, 
performance  (7t)  should  increase  as  a  function  of  exactingness  (a). 

Experimental  results.  The  above  model  and  hypotheses  were  tested  in  a  series  of  five 
experiments  —  see  Hogarth,  Gibbs,  McKenzie  &  Marquis  (in  press)  for  details.  The  experiments 
were  based  on  a  repetitive  decision  making  task  that  was  similar  to  a  single-cue  probability  learning 
task  except  that  feedback  was  not  in  the  form  of  observations  of  a  criterion  variable.  Instead, 
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feedback  was  by  way  of  “evaluation  points”  and  subjects  were  instructed  to  maximize  evaluation 
points.  Unknown  to  the  subjects,  evaluation  points  were  inversely  proportional  to  squared 
prediction  errors,  with  the  constant  of  proportionality  being  used  to  manipulate  exactingness. 
“Lenient”  environments  were  characterized  by  small  proportionality  constants  and  “exacting” 
environments  by  large  ones. 

The  main  results  of  the  experiments  were  as  follows: 

(1)  Learning  —  as  measured  by  performance  —  was  seen  to  be  an  inverted-U  shaped 
function  of  exactingness.  This  is  illustrated  in  Figure  2  below  which  shows  performance  as  a 
function  of  three  levels  of  exactingness  (lenient,  intermediate,  and  exacting)  over  two  rounds  of 
one  of  our  experiments  each  involving  30  trials  --  Prediction  1 . 

Mean 


Figure  2:  Mean  performance  (re)  by  types  of  environment  (lenient,  intermediate,  and  exacting) 

across  rounds. 
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(2)  There  was  an  interaction  between  incentives  and  exactingness.  Whereas  performance 
was  better  with  rather  than  without  explicit  monetary  incentives  in  a  lenient  environment,  this 
relationship  reversed  in  the  exacting  environment  (there  was  no  differential  effect  in  an  intermediate 
environment).  This  result  held  in  a  condition  where  the  incentive  function  was  “sharp  “  (i.e., 
distinguished  clearly  between  successful  and  unsuccessful  performance)  and  where  incentives 
were  manipulated  by  aspirations  as  opposed  to  external,  monetary  incentives  --  Prediction  2.  In 
the  case  of  a  “flat”  incentive  function  (i.e.,  which  did  not  distinguish  sharply  between  different 
levels  of  performance),  however,  incentives  had  no  differential  effect  on  performance. 

(3)  When  the  negative  effects  of  exactingness  were  mitigated  (by  providing  subjects  both 
outcome  feedback  and  evaluation  points),  performance  increased  with  exactingness  -  Prediction  3. 

(4)  Comparisons  were  made  of  performance  under  incentives  after  subjects  had  been 
trained  under  different  conditions  of  incentives  and  exactingness.  In  terms  of  performance,  the 
most  effective  methods  were  found  to  be  either  (a)  training  under  conditions  where  subjects  were 
told  to  try  to  learn.  the  task  as  opposed  to  maximize  evaluation  points  and  (b)  training  under 
intermediate  conditions  of  exactingness  (but  under  instructions  to  maximize  evaluation  points). 
The  former  method,  however,  took  much  more  time  than  the  latter. 

(5)  Questionnaires  were  constructed  to  test  subjects’  ability  to  articulate  their  understanding 
of  the  task  environment  (over  and  above  their  ability  to  predict  outcomes).  Results  of  these  tests 
were  highly  predictive  of  subjects’  individual  performance  on  the  task  and  were  also  related  to  the 
experimental  manipulations. 

We  believe  that  both  the  theory  and  experimental  results  achieved  in  this  study  are  very 
important.  In  many  decision-making  tasks  feedback  is  ambiguous  with  the  result  that  people  are 
often  ignorant  of  whether  decisions  are  taken  in  environments  that  are  “lenient”  or  “exacting”  and 
the  effects  this  can  have  on  their  ability  to  learn.  Indeed,  the  fact  that  our  experiments  generated 
such  a  complex  pattern  of  outcomes  given  the  small  nature  of  the  manipulation  (i.e.,  adjusting 
exactingness)  was  remarkable.  In  addition,  the  effects  of  incentives  on  performance  have  often 
been  puzzling,  to  say  the  least. 


17 


Technical  Reports 


Technical 


Report  No. 

19. 

Einhom,  H.  J.,  &  Hogarth,  R.  M.  (1987).  A  theory  of  diagnostic  inference: 
Contract  progress  report.  University  of  Chicago,  Center  for  Decision  Research, 

AD  #179786. 

20. 

Einhom,  H.  J.,  &  Hogarth,  R.  M.  (1987  October).  Adaptation  and  inertia  in 
belief  updating:  The  contrast-inertia  model.  University  of  Chicago,  Center  for 
Decision  Research. 

21. 

Hogarth,  R.  M.,  &  Einhom,  H.  J.  (1988  January).  Venture  theory:  A  model 
of  decision  weights.  University  of  Chicago,  Center  for  Decision  Research, 

AD  #A  194809. 

22. 

Hogarth,  R.  M.  (1988  May).  Ambiguity  and  competitive  decision  making:  Some 
implications  and  tests.  University  of  Chicago,  Center  for  Decision  Research, 

AD  #A  196149. 

23. 

Hogarth,  R.  M.,  &  Kunreuther,  H.  (1988  November).  Pricing  insurance  and 
warranties:  Ambiguity  and  correlated  risks.  University  of  Chicago,  Center  for 
Decision  Research,  AD  #A203773. 

24. 

Hogarth,  R.  M„  Gibbs,  B.  J.,  McKenzie,  C.  R.  M„  Marquis,  M.  A.  (1989  June). 
Learning  to  make  decisions:  When  incentives  help  and  hinder.  University  of 
Chicago,  Center  for  Decision  Research,  AD  #A210185. 

25. 

Hogarth,  R.  M.,  &  Kunreuther,  H.  (1990  May).  Decision  making  under 
uncertainty:  The  effects  of  role  and  ambiguity.  University  of  Chicago,  Center 
for  Decision  Research,  AD  #A225771. 

26. 

Kunreuther,  H.,  &  Hogarth,  R.  M.  (1990  May).  How  does  ambiguity  affect 
insurance  decisions?  University  of  Chicago,  Center  for  Decision  Research, 

AD  #A225707. 

18 


Publications 


Einhom,  H.  J.,  &  Hogarth,  R.  M.  (1987).  Decision  making:  Going  forward  in  reverse. 

Harvard  Business  Review,  65(1),  66-70. 

Goldstein,  W.  M.,  &  Einhom,  H.  J.  (1987).  Expression  theory  and  the  preference  reversal 
phenomena.  Psychological  Review,  94(2),  236-254. 

Hogarth,  R.  M.  (1987).  Judgement  and  choice:  The  psychology  of  decision.  (2nd  ed.)  Chichester, 
UK:  Wiley. 

Hogarth,  R.  M.,  &  Reder,  M.  W.  (Eds.)  (1987).  Rational  choice:  The  contrast  between 
economics  and  psychology.  Chicago:  University  of  Chicago  Press. 

Simon,  H.  A.,  et  al.  (1987).  Decision  making  and  problem  solving.  Interfaces,  17(5),  11-31. 

(Reprinted  from  "Report  of  the  Research  Briefing  Panel  on  Decision  Making  and  Problem 
Solving"  in  Research  Briefings  1986,  Washington,  DC:  National  Academy  Press.) 

Einhom,  H.  J.  (1988).  Diagnosis  and  causality  in  clinical  and  statistical  prediction.  In  D.  C.  Turk 
&  P.  Salovey  (Eds.),  Reasoning,  inference,  and  judgment  in  clinical  psychology,  51-70. 
New  York:  The  Free  Press. 

Einhom,  H.  J.,  &  Hogarth,  R.  M.  (1988).  Decision  making  under  ambiguity:  A  note.  In 

B.  Munier  (Ed.),  Risk,  decision  and  rationality,  327-336.  Dordrecht,  Holland:  D.  Reidel 
Publishing  Company. 

Hogarth,  R.  M.,  &  Klayman,  J.  (1988).  Hillel  J.  Einhom  (1941-1987).  [Obituary],  American 
Psychologist,  43,  656. 

Hogarth,  R.  M.,  &  Makridakis,  S.  (1988).  Consider  the  principal  finding:  A  reply  to  William  T. 
Ross.  Management  Science,  34(5),  672-673. 

de  Wit,  H.,  Hogarth,  R.  M.,  Koehler,  J.  J.,  &  Luchins,  D.  J.  (1989).  Effects  of  diazepam  on  a 
belief-updating  task.  Psychological  Reports,  64,  219-226. 

Goldstein,  W.  M.,  &  Einhom,  H.  J.  (1989).  Expression  theory  and  the  measurement  of 
apparently  labile  values.  Annals  of  Operations  Research,  19,  51-78. 

Hogarth,  R.  M.  (1989a).  On  combining  diagnostic  'forecasts':  Thoughts  and  some  evidence. 
International  Journal  of  Forecasting,  5,  593-597. 

Hogarth,  R.  M.  (1989b).  Ambiguity  and  competitive  decision  making:  Some  implications  and 
tests.  In  P.  C.  Fishbum  &  I.  H.  La  Valle  (eds.),  Annals  of  Operations  Research,  19,  31- 
50. 

Hogarth,  R.  M.  (1989c).  Transgressive  action  and  inaction:  The  need  to  study  both.  Acta 
Psychologica,  70, 65-69  (invited  commentary). 


19 


Hogarth,  R.  M.,  &  Kunreuther,  H.  (1989).  Risk,  ambiguity,  and  insurance.  Journal  of  Risk 
and  Uncertainty ,  2,  5-35. 

Klayman,  J.,  &  Hogarth,  R.  M.  (1989).  Does  ECHO  explain  explanation?  A  psychological 

perspective.  The  Behavioral  and  Brain  Sciences,  12(3),  478-479  (invited  commentary). 

Hogarth,  R.  M.  (Ed.)  (1990).  Insights  in  decision  making:  A  tribute  to  HillelJ.  Einhorn. 
Chicago:  The  University  of  Chicago  Press. 

Hogarth,  R.  M.,  &  Einhorn,  H.  J.  (1990a).  Venture  theory:  A  model  of  decision  weights. 
Management  Science,  36(7),  780-803. 

Hogarth,  R.  M.,  &  Einhorn,  H.  J.  (1990b  May).  Order  effects  in  belief  updating :  The  belief- 
adjustment  model.  University  of  Chicago,  Graduate  School  of  Business,  Center  for 
Decision  Research. 

Hogarth,  R.  M.,  Gibbs,  B.  J.,  McKenzie,  C.  R.  M.,  &  Marquis,  M.  A.  (in  press). 

Learning  from  feedback:  Exactingness  and  incentives.  Journal  of  Experimental 
Psychology:  Learning,  Memory,  and  Cognition. 

Hogarth,  R.  M.,  &  Kunreuther,  H.  (1990  October).  Pricing  insurance  and  warranties: 

Ambiguity  and  correlated  risks.  University  of  Chicago,  Center  for  Decision  Research. 

Hogarth,  R.  M„  &  Kunreuther,  H.  (in  press).  Decision  making  under  uncertainty:  The  effects 
of  role  and  ambiguity.  In  F.  Heller  (Ed.),  Decision  making  and  leadership.  Cambridge, 
England:  Cambridge  University  Press. 

Kunreuther,  H.,  &  Hogarth,  R.  M.  (in  press).  How  does  ambiguity  affect  insurance  decisions? 
In  G.  Dionne  (Ed.),  Insurance  economics.  Boston:  Kluwer. 


References  in  report  not  included  in  above  lists 


Hogarth,  R.  M.  (1986).  Generalization  in  decision  research:  The  role  of  formal  models.  IEEE 
Transactions  on  Systems,  Man,  and  Cybernetics,  SMC-16,  439-449. 

Kahneman,  D„  &  Tversky,  A.  (1979).  Prospect  theory:  An  analysis  of  decision  under  risk. 
Econometrica,  47,  263-291. 


20 


Ph.  D.  Theses 


Gibbs,  B.  J.  (1991,  expected).  Variable  tastes:  Self-manipulation  of  stimulus  valuations. 
Unpublished  doctoral  dissertation,  University  of  Chicago. 

Koehler,  J.  J.  (1989).  Judgments  of  evidence  quality  among  scientists  as  a  function  of  prior 
beliefs  and  commitments.  Unpublished  doctoral  dissertation,  University  of  Chicago 

Maines,  L.  (1990).  Data  redundancy  and  consensus  forecasting  judgments. 

Unpublished  doctoral  dissertation,  University  of  Chicago 


Acknowledgements  and  Scientific  Personnel 

The  work  reviewed  in  this  report  would  not  have  been  possible  without  the  effort  of  several 
co-workers.  I  would  particularly  like  to  thank  Charlesetta  Wren  for  her  administrative  assistance. 
In  addition,  the  following  graduate  students  made  important  contributions  to  various  aspects  of  the 
research:  Brian  Gibbs,  Jonathan  Koehler,  Craig  McKenzie,  and  Alison  Marquis.  Finally,  several 
colleagues  —  notably  Howard  Kunreuther,  William  Goldstein,  Joshua  Klayman,  and  George 
Loewenstein  -  provided  much  help  and  encouragement. 


TECHNICAL  REPORTS  DISTRIBUTION  LIST 


Mr.  Charles  Bates 
Director,  HE  Division 
USAF  AAMRL/HE 
WPAFB,  OH  45433 

Mr .  Norman  Beck 

Naval  Underwater  Systems 

Center 

Combat  Control  Systems 
Department  Code  221 
Newport,  RI  02840 

Dr.  Kenneth  R.  Boff 
AAMRL/HE 

WPAFB,  OH  45433 

Mr.  Luiz  Cabral 

Naval  Underwater  Sys.  Ctr. 

Code  2212 

Bldg.  1171/1 

Newport,  RI  02841 

Cdr.  Robert  C.  Carter  USN 
Naval  Research  Laboratory 
Code  5532 

Washington,  DC  20375-5000 


Dr.  Marvin  S.  Cohen 
Dec.  Sci.  Consortium,  Inc. 
1895  Preston  White  Drive 
Suite  300 

Reston,  VA  22091 

Dr.  Stanley  C.  Collyer 
Office  of  Naval  Technology 
Code  222 

800  N.  Quincy  Street 
Arlington,  VA  22217-5000 

Defense  Tech.  Info.  Center 
Cameron  Station,  Bldg  5 
Alexandria,  VA  22314 

(2  copies) 


Naval  Ocean  Systems  Center 
Human  Factors  Support  Group 
1411  South  Fern  Street 
Arlington,  VA  22202-2896 

Dr.  Alfred  R.  Fregly 
AF  Office  of  Sci.  Res. 

Life  Sciences  Directorate 
Bldg.  410 

Bolling  AFB,  DC  20332-6448 

Mr.  Jeffrey  Grossman 
Naval  Ocean  Systems  Center 
Code  4402 
Bldg.  334 

San  Diego,  CA  92152 

Dr.  Thomas  E.  Hanna 
Naval  Sub  Med  Res  Lab 
Submarine  Systmes  Dept. 
Naval  Submarine  Base 
Groton,  CT  06349-5900 

Dr.  Edgar  M.  Johnson 
Technical  Director 
US  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333-5600 

Dr.  Milton  S.  Katz 
US  Army  Research  Institute 
ARIESCO 
Box  65 

FPO ,  NY  09510-1500 

Prof.  David  L.  Kleinman 
Electrical  Engineering  & 
Computer  Science  Dept. 
University  of  Connecticut 
Storrs,  CT  06268 

Dr.  William  Rouse 
Search  Technology,  Inc. 

4725  Peachtree 
Corners  Circle 
Suite  200 

Norcross,  GA  30092 

Capt  T.  E.  Jones,  MSC,  USN 
Aviation  Medicine  &  Human 
Performance  (Code  404) 

Naval  Medical  R  &  D  Cmd 
National  Capital  Region 
Bethesda ,  MD  21814-5044 


Dr.  Michael  Kaplan, 

Director,  Office  Basic  Res 
US  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333-5600 

Dr.  Alexander  Levis 
Department  of  Electrical  and 
Computer  Engineering 
George  Mason  University 
4400  University  Drive 
Fairfax,  Virginia  22030 

Dr.  Richard  Loutitt 
National  Science  Foundation 
Division  of  Behavioral  & 
Neural  Sciences 
1800  G  Street,  N.  W. 
Washingtoft,  DC  20550 

Office  of  Naval  Research 
Perceptual  Science  Program 
Code  114 2 PS 
800  N.  Quincy  Street 
Arlington,  VA  22217-5000 
(3  copies) 

Naval  Aerospace  Medical 
Research  Laboratory 
Sensory  Division  Code  23 
Pensacola,  FL  32508 

Naval  Air  Systems  Command 
Crew  Station  Design 
NAVAIR  5313 
Washington,  DC  20361 


Commanding  Officer 
Navy  Health  Research  Ctr. 
P.0.  Box  85122 
San  Diego,  CA  92138 

Technical  Director 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Director 

Technical  Information  Div. 
Code  2627 

Naval  Research  Laboratory 
Washington,  DC  20375-5000 


US  Naval  Test  Center 
Aircrew  Systems  Branch 
Systems  Engineering  Test 
Directorate 

Patuxent  River,  MD  20670 


Naval  Weapons  Center 
Human  Factors  Branch 
Code  3152 

China  Lake,  CA  93555 

Dr.  Michael  Letsky 
Office  of  the  Chief  of 
Naval  Operations 
OP-01B7 

Washington,  DC  20350 

Dr.  Jesse  Orlansky 
Inst,  for  Defense  Analyses 
1801  N.  Beauregard  Street 
Alexandria,  VA  22311 

Prof.  John  W.  Payne 
Fuqua  School  of  Business 
Ctr.  for  Decision  Studies 
Duke  University 
Durham,  NC  27706 

Dr.  E.  Salas 
Head,  Human  Factors  Div. 
Naval  Training  Systems  Ctr 
12350  Research  Parkway 
Orlando,  FL  32826-3224 

Prof  Frank  Snyder 
Naval  War  College 
Operations  Department 
Newport,  RI  02841 

Dr.  H.  Wallace  Sinaiko 
Manpower  Research  & 
Advisory  Services 
Smithsonian  Institution 
801  N.  Pitt  St.  Suite  120 
Alexandria,  VA  22314-1713 

Mr.  James  G.  Smith 
Office  of  Naval  Research 
Code  121 

800  N.  Quincy  Street 
Arlington,  VA  22217-5000 


Dr.  Douglas  Towne 
University  of  Southern 
California 

Behavioral  Technology 
Laboratory 

1845  South  Elena  Avenue 

Fourth  Floor 

Redondo  Beach,  CA  90277 

Dr.  Harold  p.  van  Cott 
Committee  on  Human  Factors 
NAS-National  Res.  Council 
2101  Constitution  Ave,  N.W. 
Washington,  DC  20418 

Dr.  John  Weisz 
Technical  Director 
US  Army  Human 
Engineering 
Laboratory 
Aberdeen 

Proving  Grounds 
MD  21005 


Prof.  Christopher  D.  Wickens 
Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61801 

Dr.  Wayne  Zachary 
CHI  Systems  Inc 
Bethlehem  Pike  at  Sheble  Lane 
Spring  House,  PA  19477 


